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Foreword 



The National Center for Public Policy and Higher Education undertook 
a study to determine the de facto standards, or the knowledge and skills 
required to successfully enroll in college-level classes at California community 
colleges. While California community colleges are open to all students eligible 
and able to benefit, most students must demonstrate that they are ready for 
college-level academic work by passing the college's placement exam once 
they enroll. Because California does not explicitly define college readiness 
standards for high school students, the placement exams function as the de 
facto entry-level standards for higher education. The community colleges 
have academic standards; high schools, teachers, students, and parents have 
just not been clearly informed of them. 

Based on their performance on the placement tests, the vast majority of 
students enrolling in a community college need remedial coursework in math 
and nearly half of them need it in English. 

Over 94 different placement assessments were given to entering students 
last year at the community colleges. This study identifies the standards 
represented by the myriad placement tests in English language arts and 
mathematics. It then compares these standards to what high school students 
are expected to know in the 11th grade in order to determine if what we are 
expecting in high school matches what students need to know to take credit- 
bearing courses at the community college. This issue is critical for high school 
students, since the vast majority of students pursuing higher education in 
California will begin at a community college. It is also critical for California, 
since the future workforce depends upon the success of community colleges 
in educating these students. 

The importance of this issue is reinforced by a recent national report 
by Achieve, Inc., Aligned Expectations? A Closer Look at College Admissions 
and Placement Tests, which concludes that the alignment of high school 
coursework and assessments with those in higher education is a necessary 
step in preparing more students to successfully enroll in and complete 
certificate and degree programs. In fact, the findings of the Brown and 
Niemi study described in this report and the subsequent recommendations 
by the National Center for Public Policy and Higher Education point to the 
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importance of linking higher education placement exams to high standards 
and assessments of high school students in math and English language arts. 

The National Center is indebted to Richard Brown and David Niemi for 
their leadership and analysis of this important issue. We are also grateful to 
our lead advisors on this project: Michael Kirst, senior fellow at the National 
Center and professor of education at Stanford University and Peter Ewell, 
vice president at the National Center for Higher Education Management 
Systems. The National Center is also grateful to the College Board and ACT 
who shared their test blueprints with the National Center for this analysis. We 
are also grateful to the California Community Colleges Chancellor's Office 
and to the California Department of Education for information made available 
to complete this analysis. 

Also contributing their insight into our analysis was a national advisory 
group made up of the following: chair of the advisory group, Michael Usdan, 
senior fellow. National Center for Public Policy and Higher Education and 
senior fellow at the Institute for Educational Leadership; Steve Bruckman, 
executive vice chancellor, California Community Colleges; Andrea Conklin 
Bueschel, research scholar. The Carnegie Eoundation for the Advancement 
of Teaching; Pamela Burdman, program officer. Education, The William and 
Elora Hewlett Eoundation; Peter T. Ewell, vice president. National Center for 
Higher Education Management Systems; Marlene L. Garcia, vice chancellor of 
governmental relations and external affairs division, California Community 
Colleges; Robert H. McCabe, senior fellow. League for Innovation in the 
Community College; Brad Phillips, executive director, Cal-PASS; Anna 
Rothman, fellow. Senate Office of Research, California Legislature; Nancy 
Shapiro, associate vice chancellor. University System of Maryland; Nancy 
Shulock, executive director. Institute for Higher Education Leadership and 
Policy, California State University, Sacramento; and Abdi Soltani, executive 
director. Campaign for College Opportunity. 

And contributing from the National Center for Public Policy and Higher 
Education were Jonathan Eelder, policy analyst; Valerie Lucas, assistant to the 
vice president; and Mikyung Ryu, senior policy analyst. 

Many community college leaders also assisted Brown and Niemi in their 
evaluation of the placement exams. They include: Daniel Bahner, title V 
activity director/ student interventions, Crafton Hills College; Kirk Gorrie, 
associate faculty member, Irvine Valley College; Valerie Henry, lecturer. 
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University of California, Irvine; Keith Howard, assistant professor, Rossier 
School of Education, University of Southern California; Mickey Jackson, 
professor emeritus. Golden West Community College; Micah Jendian, English 
instructor, Grossmont Gollege; Leanne Maunu, associate professor, Palomar 
Gollege; Tim Thayer, math teacher, Mira Loma High School/ San Juan Unified 
School District; and Julio Villarreal, former instructor, San Diego Gity College. 

The National Center would like to thank The James Irvine Eoundation for 
sponsoring this special project. We welcome the reactions of readers to this 
report. 



Patrick M. Callan 
President 



Joni E. Finney 
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Introduction 



The California Community College system is the largest system of higher 
education in the world. Its 72 districts and 109 campuses served more than 2.5 
million students in the 2005-2006 academic year. According to the California 
Community Colleges Chancellor's Web site (www.cccco.edu), the "...primary 
missions of the Colleges are to offer academic and vocational education at the 
lower division level for both recent high school graduates and those returning 
to school. Another primary mission is to advance California's economic 
growth and global competitiveness through education, training, and services 
that contribute to continuous workforce improvement." In addition, the 
Strategic Plan for the California Community Colleges lists the missions of 
the community college system as transfer education, basic skills and English 
language proficiency instruction, economic and workforce development, 
lifelong learning, and providing associate's degrees and certificates (California 
Community Colleges, 2006). The Strategic Plan also lists several strategic 
goals for the community college system. These include: increase college 
awareness and access; promote student success and readiness; strengthen 
partnerships for economic and workforce development; improve system 
effectiveness; and provide enhanced resources. This study focuses on only one 
aspect of these multiple objectives and that is the goal of promoting student 
success and readiness by evaluating the system of evaluating and placing 
students into community college courses. 

As a result of the open access policy in place at community colleges in 
California, not all of the students who enroll are prepared for the academic 
rigors of college-level work. A recent study from the Institute for Higher 
Education Leadership & Policy concludes that the open access policies of 
California's community colleges have succeeded in enhancing enrollments, 
but have had the unintended consequence of inhibiting college completion 
(Shulock & Moore, 2007). They argue that since only one in four degree- 
seeking community college students actually earns a certificate or degree, 
transfers to a four-year university, or achieves some combination of those 
outcomes within six years of enrolling in a community college, policies should 
be changed to encourage better educational outcomes rather than simply 
focusing on allowing students to enroll. 

Additional data offer further support to the argument that many students 
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enrolling from high school into community colleges in California are 
unprepared for college-level coursework. For example, the strategic planning 
research report entitled. Environmental Scan: A Summary of Key Issues Facing 
California Community Colleges Pertinent to the Strategic Planning Process stated, 
"A recent survey of California community college placement test results 
indicated that only about 9% of students place in transfer level math and 
about 27% of students place in transfer level English. . .over 70% of students 
place in remedial math and 42% place in remedial English" (Research and 
Planning Group for California Community Colleges, 2005, p. 6). This means 
that the vast majority of students are initially placed in courses for which 
they will not receive credit at a California State University or University of 
California campus if and when they choose to transfer. 

Having such large numbers of students take remedial courses is not 
without consequence. Students who start out in the remedial levels of math 
and reading courses have limited probability of attempting transfer level 
courses at the community college. The likelihood of attempting a transfer 
level English course after beginning in a reading fundamentals course at the 
community college is only 25% (Research and Planning Group for California 
Community Colleges, 2005). The numbers are more dismal for mathematics. 
The likelihood of taking a transfer level math course after starting in a basic 
level math course is only 10%. The Research Group report says, "Empirical 
evidence suggests that those who begin at the lowest levels of basic skills 
are unlikely to achieve a degree or transfer to a university" (p. 6). Given this 
evidence, there is reason to be concerned about the fact that one in every three 
students in the community colleges enrolls in a basic skills class. 

Placement Testing in Calieornia Community Colleges 

It is clear that many students entering the community college campuses are 
not prepared for college-level coursework. As data in the table on page 3 
show, for students enrolled in 2005-2006 at California community colleges, 
of the more than one million students between the ages of 18 and 24 enrolled 
in credit-bearing courses, over 800,000 were directed to placement testing 
services during their time at the community college campus, of which 475,000 
actually participated in placement testing. Another 340,000 students were 
directed to placement testing but did not take a placement test. 

Determining which of the thousands of students at a given campus need 
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remediation is a difficult and challenging task. This task is not undertaken 
consistently across the state. While some campuses exempt students from 
placement testing based on previous collegiate coursework or Advanced 
Placement test scores from high school, there is no uniform placement 
testing process throughout the community college system. Each of the 
109 community college campuses determines for itself which placement 
examinations to use and what the proficiency 
cut-scores will be for each test and placement 
decision. This flexibility in placement 
testing practices is not limited to California. 

According to a report on state policies 
regarding placement testing across the 
country, only "a small number of states have 
established minimum passing scores for 
entry into general education without referral 
to developmental education; more often such 
decisions are left to institutional discretion" 

(Prince, 2005, p. 3). 

As a result of this varied approach to placement testing, nearly 100 
different examinations are used for placement purposes across the state. 

The Chancellor's Office official Web site reports 94 assessments that were 
administered to students who were enrolled in the 2005-2006 academic year 
and more are on the list of assessments approved by the Chancellor's Office 
for use as placement instruments. This reflects the tremendous variability of 
placement testing practices. Three dozen second-party assessments gained 
approval, while more than one hundred other approved tests are listed as 
"locally developed and locally managed assessment instruments." Such 
variety yields inconsistency throughout the community college system not 
only in test content but also in levels of expected proficiency within a given 
subject area domain. Thus, it is unsurprising that students leaving high school 
for the community college campus are unaware of what it takes to be deemed 
adequately prepared for college-level coursework. 

Placement testing volume in California community colleges is extensive. 
The volume of testing is inconsistent across assessments, however. Several 
placement exams were given to over 100,000 students, while others 
were presented to only a handful of students. Some of the assessments 
are commercially developed with extensive documentation to evaluate 



Table 1. 

Flow of Use of Community College Placement Services 



Number of 

Placement Assessment Services Students 



Enrolled in Credit-Bearing Courses 


1,050,000 


Exempted from Placement Testing 


235,000 


Directed to Placement Testing 


815,000 


Did Not Receive Placement Testing 


340,000 


Received Placement Testing 


475,000 



NOTE: This table represents assessment services for placement provided to students aged 
18-24 enrolled tor Annual 2005-2006. These services may have been provided at 
any time during their academic career up to and including Annual 2005-2006. Data 
provided by the California Community Colleges Chancellor’s Office Web site 
(www.cccco.edu). 
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their psychometric characteristics (although validation for community 
college placements is not always included). The most commonly used 
assessments are commercially developed and marketed assessments such 
as the ACCUPLACER CPTs by the College Board and the COMPASS tests 
developed by ACT. These two test batteries account for more than half 
of all placement testing in California community colleges. Other tests in 
widespread use include the Math Diagnostic Testing Program (MDTP), which 
was originally developed by faculty from the UC and CSU systems (current 
test development activities also include faculty from the community colleges), 
and the California Test for English Placement (CTEP), which was developed 
by California community college faculty. 

Other placement exams in use across the community college campuses 
are locally developed and lack the requisite documentation to fully gauge the 
quality of the instruments, making judgments about their utility speculative. 
In some instances, such as at American River College in Sacramento, the 
community colleges deploy a self-placement approach wherein students 
are provided access to instruments to assess themselves and make course 
placement decisions based on those assessments. According to the American 
River College institutional research office, in these self-placement scenarios, 
validation of the self-placement assessment is not required under existing 
regulations. One thing is certain: the process for placement testing in 
California community colleges is inconsistent across campuses and student 
groups. What is tested and what constitutes proficiency or preparedness for 
college-level, credit-bearing courses varies dramatically across the system. 

In addition to testing for proficiency in mathematics and English language 
arts (ELA), community colleges use different English as Second Language 
(ESL) placement exams for the placement of English learners into various 
ESL courses. Recent research funded by the University of California All 
Campus Consortium on Research for Diversity (UC/ACCORD) indicates 
that campuses vary widely in how they use ESL placement exams relative to 
regular English placement exams (Bunch, 2006). In research conducted with 
15 community colleges. Bunch found that how campuses determine whether 
to advise students to take the ESL placement or the regular English placement 
varies across campuses using determinants, such as "self-evident" descriptors 
like "students for whom English is a second language," or comparisons with 
other languages, like "students whose strongest language is not English." 

He notes that, for the 15 campuses he studied, the criteria for determining 
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whether a student should take an ESL placement or a regular English 
placement test did not explicitly include scores from either the California 
English Language Development Test (CELDT), which is a test of English 
proficiency given in high schools, high school language designations, or 
scores from any other mandated high school assessment. 

Bunch also found that while the systemwide policy clearly states that 
students can choose whether to take an ESL placement test or a regular 
English placement test, the campuses themselves do not make clear that 
students have this choice. Rather, they provide guidance as to who should 
take ESL tests or the regular English placements based on varied criteria. 
Given that the choice between these two types of placement tests determines 
which path toward credit-bearing English courses the student undertakes, 
the choice is important. Students take either the ESL placement or the regular 
English placement, but not both. Those taking the ESL placement are placed 
in one of several noncredit-bearing ESL courses. Those students taking a 
regular English placement exam may be referred to take the ESL placement 
test, but are generally placed in either a remedial (non-ESL) English course 
or nonremedial (non-ESL) English course. Taking the ESL placement can 
severely delay access to credit-bearing English courses. As Bunch accurately 
points out, because many language minority students are already at risk of 
dropping out, this initial determination of which placement test they take 
may well impact their college persistence and completion. 

Efficacy of Placement Testing 

With so much placement testing in the California Community College System, 
it is reasonable to ask whether such testing and placement are having a 
positive impact on student learning and student progression through college. 
The research on the efficacy of placement testing is mixed. In a review of 
research on the predictive validity of placement tests, Armstrong (1999) found 
that most of the studies relating community college placement test scores 
to course grades resulted in small or modest relationships. Armstrong cites 
numerous studies and provides illustrative examples of instances at specific 
community colleges in California where the placement tests were related 
to course outcomes, although few predictive validity coefficients exceeded 
the 35 threshold recommended by the Chancellor's Office. Armstrong 
summarizes the literature by saying, "most of these studies have found the 
validity coefficients to be low to moderate" (p. 82). Armstrong argues that 
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other variables, such as the grade expected in the course or grade received 
in the student's most recent mathematics course, had approximately the 
same predictive validity as math placement test scores. Similar findings were 
noted in predicting performance in English courses. Armstrong further cites 
research that found low predictive validity coefficients for the ASSET writing 
test and ASSET reading test on predicting English composition grades. 

In another study, Armstrong (1994) showed that variation in student 
course grades was primarily a function of instructor grading variability and 
not as much influenced by placement test results. He recommends that any 
analytic model trying to predict student course outcomes should include 
instructors as well as placement test scores in the model to account for the 
greatest amount of variation in student outcomes. Armstrong also points 
out that students identified through placement exams as "unprepared" or 
"ineligible" for credit-bearing courses, but nonetheless took the courses, 
actually performed relatively well in those courses for which they were 
deemed ineligible. He argues, "The repeated finding that ineligible students 
were generally 70%-80% as successful as eligible students suggests strongly 
that student access may be unfairly denied and that many students capable 
of success are not given the chance to try. Inaccurate referral to lower level 
courses increases the opportunity cost for the students, and misallocates the 
educational resources of the college" (p. 208-209). 

Providing strong predictive validity evidence for placement exams is not 
easy, particularly since such studies typically yield low predictive validity 
coefficients. Yet according to matriculation pohcies, each community college 
campus is required to periodically provide validity evidence of whatever 
placement system they employ, resulting in much expended energy and 
resources at the campus level. The Research and Planning Group for California 
Community Colleges laments this requirement, noting, "Limited research time 
can be better spent working with faculty and support staff on strategies that 
relate to student success, student learning outcomes, and persistence rather 
than validating a plethora of tests for 109 colleges" (2004, p. 1). 

In addition to providing predictive validity coefficients, another way of 
determining whether placement testing is effective is to look at the success 
rate of students placed into specific courses. Ideally, if students are properly 
placed into courses for which they are most suited, they should perform 
successfully in those courses. Success in a course can be measured as the 
proportion of students obtaining a grade of C or better or obtaining a CR 



High School and Community College Assessments 



grade indicating that the student Table 2. 

earns credit for the course. While Rate of Success in Community Coiiege Courses, by Ethnicity 



the overall success rate in California 
community college courses is over 
66%, there are extreme variations 
across ethnicity and course type. 
White and Asian students have the 
highest rate of success in community 
college courses at 71%, while the 
success rate for Hispanic students is 
just under the overall success rate at 
61%. African-American community 
college students are less successful in 
community college courses, with an 
overall success rate of 54%. However, 



Ethnicity 


Totai 

Enroiiments 


Number of 
Successes 


Success 
Rate (%) 


African-American 


293,609 


159,324 


54.26 


American indian/Aiaskan Native 


34,922 


21,802 


62.43 


Asian 


437,889 


311,849 


71.22 


Fiiipino 


136,129 


91,416 


67.15 


Hispanic 


1,016,061 


623,565 


61.37 


Other Non-White 


75,481 


49,797 


65.97 


Pacific isiander 


29,306 


17,932 


61.19 


Unknown 


245,525 


168,042 


68.44 


White Non-Hispanic 


1,329,594 


946,476 


71.19 


Grand Totai 


3,598,516 


2,390,203 


66.42 



NOTE: This table represents student success by ethnicity in all courses in spring 2006. Success is 

characterized by obtaining a grade of A, B, C, or CR. The denominator for the success equation is 
the number of enroiiments with grade of A,B,C,D,F,CR,NC,W,i. Data provided by the California 
Community Colleges Chancellor’s Office Web site (www.cccco.edu). 



these rates are across all courses. When the rates are broken out by the type 
of course (e.g., Basic Skills, Non-Basic Skills, or Pre-Collegiate Basic Skills) 
differences in success across course type become apparent. 



For Basic Skills and Pre-Collegiate Basic Skills courses (non-degree credit 
courses, commonly called "remedial"), the rates are much lower than the 
rate for Non-Basic Skills courses, as shown in subsequent tables. For white 
students, the success rate drops from 72% in Non-Basic Skills courses to 
only 58% and 62% for Basic Skills and Pre-Collegiate Basic Skills courses, 
respectively. Likewise, the rates for African-American and Hispanic students 
are lower in the remedial courses than in the nonremedial courses. The 



success rate for African-Americans Table 3 . 

is 40% in Basic Skills and 41 % in Rate of Success in Non-Basic Skiiis Courses, by Ethnicity 



Pre-Collegiate Basic Skills courses 
as opposed to 56% in Non-Basic 
Skills courses. The same pattern 
holds for Hispanics, where the rate 
for the Basic Skills courses is 38% 
compared to 62% for Non-Basic 
Skills courses. Interestingly, Hispanic 
students had better success in the 
Pre-Collegiate Basic Skills courses 
than they did in the Basic Skills 
courses, with a success rate of 54%. 



Ethnicity 


Total 

Enrollments 


Number of 
Successes 


Success 
Rate (%) 


African-American 


265,854 


147,889 


55.63 


American indian/Aiaskan Native 


32,689 


20,633 


63.12 


Asian 


397,509 


288,567 


72.59 


Fiiipino 


130,085 


87,922 


67.59 


Hispanic 


910,041 


569,364 


62.56 


Other Non-White 


70,993 


47,267 


66.58 


Pacific isiander 


27,582 


17,108 


62.03 


Unknown 


233,146 


161,955 


69.47 


White Non-Hispanic 


1 ,278,260 


914,652 


71.55 



NOTE: This fable represents student success by ethnicity and Basic Skills status in courses in spring 
2006. Success is characterized by obtaining a grade of A, B, C, or CR. The denominator for the 
success equation is the number of enrollments with grade of A,B,C,D,F,CR,NC,W,I. Data provided by 
the California Community Colleges Chancellor’s Office Web site (www.cccco.edu). 
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Table 4. 



From these data on student success 



Rate of Success in Basic Skiiis Courses, by Ethnicity 

Total Number of Success 



Ethnicity Enrollments Successes Rate (%) 



African-American 


2,774 


1,111 


40.05 


American indian/Aiaskan Native 


354 


202 


57.06 


Asian 


8,914 


1,895 


21.26 


Fiiipino 


831 


435 


52.35 


Hispanic 


18,003 


6,753 


37.51 


Other Non-White 


474 


229 


48.31 


Pacific isiander 


196 


77 


39.29 


Unknown 


3,440 


998 


29.01 


White Non-Hispanic 


7,228 


4,192 


58.00 


NOTE: This table represents student success by Ethnicity and Basic Skills status in courses in spring 
2006. Success is characterized by obtaining a grade of A, B, C, or CR. The denominator for the 
success equation is the number of enrollments with grade of A,B,C,D,E,CR,NC,W,I. Data provided by 
the California Community Colleges Chancellor’s Office Web site (www.cccco.edu). 




Table 5. 






Rate of Success in Pre-Collegiate Basic Skills Courses, by Ethnicity 


Ethnicity 


Total 

Enrollments 


Number of 
Successes 


Success 
Rate (%) 


African-American 


24,981 


10,324 


41.33 


American Indian/Alaskan Native 


1,879 


967 


51.46 


Asian 


31,466 


21,387 


67.97 


Filipino 


5,213 


3,059 


58.68 


Hispanic 


88,017 


47,448 


53.91 


Other Non-White 


4,014 


2,301 


57.32 


Pacific Islander 


1,528 


747 


48.89 


Unknown 


8,939 


5,089 


56.93 


White Non-Hispanic 


44,106 


27,632 


62.65 



NOTE: This table represents student success by Ethnicity and Basic Skiiis status in courses in spring 
2006. Success is characterized by obtaining a grade of A, B, C, or CR. The denominator for the 
success equation is the number of enroiiments with grade of A,B,C,D,E,CR,NC,W,i). Data provided 
by the Caiifornia Community Coiiege Chanceiior’s Office Web site (www.cccco.edu). 



in Basic Skills and Pre-Collegiate 
Basic Skills courses, it is difficult to 
argue that students are placed into 
courses that are well matched to 
their ability or achievement levels. 
This is particularly true for African- 
American and Hispanic students, 
where the majority of students 
placed in Basic Skills courses do not 
achieve success in those courses. 
Moreover, conditions are not 
expected to improve any time soon. 
As the community college research 
group states, "With an increase in 
the number of students needing 
remediation, overall success rates 
may begin to decline unless there is 
a commensurate increase in support 
services and/or improved methods 
of basic skills education" (Research 
and Planning Group for California 
Community Colleges, 2005, p. 6). 

In summary, the inconsistent 
and varied processes for placement 
testing across the California 
Community College system result in 



uneven effects and unnecessary burdens on the community college campuses. 



The evidence on the predictive validity, utility, or effectiveness of placement 
tests currently in use is not compelling, and the success rate of students 



in community college courses, particularly in Basic Skills courses, is not 



overwhelming. The research group advising community college Chancellor 
Mark Drummond summarized it this way: 



". . .after 18 years' experience with the colleges' process 
and 18 years perspective on the outcomes, it is the belief of the 
RP Group that California's community college students are 
not well served by the current approach, the implementation 
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of which has resulted in great inconsistency in measures, 
processes, and placement outcomes from college to college. 

Those inconsistencies, in turn, have created unnecessary 
barriers for students entering the community colleges and 
for students attempting to transfer within our system. The 
existence of 109 assessment processes has made it difficult, 
if not almost impossible, to consider aligning placement in 
community college basic skills courses with the exit standards 
of the secondary system" (Research and Planning Group for 
California Community Colleges, 2004, p. 1). 

Disjuncture Between Secondary and Community College Education 

Clearly, a disjuncture between the secondary and community college 
education systems in California exists. Much has been written about the poor 
transition for students between secondary and postsecondary educational 
systems in the United States (see for example, Conley, 2003a; Conley 2003b; 
Kirst & Bracco, 2004; Kirst & Venezia, 2004; Smith & Wertlieb, 2005; Venezia, 
Kirst, & Antonio, 2003). Some argue that the transition is affected by student 
expectations (Karp, Holmstrom & Gray, 1998; Smith & Wertlieb, 2005) and 
social concerns (Holmstrom, Karp, & Gray, 2002; Tan, 1996). Others point to 
more structural aspects, such as a lack of formal linkages between secondary 
and postsecondary systems in terms of governance structures (Conklin, 2005) 
and accountability, information, and data systems (Kirst & Bracco, 2004; 
Venezia, Kirst, & Antonio, 2003). The disconnection between high school 
and college is also reflected in academic subject matter content. In fact, the 
most recent Quality Counts report from Education Week indicates that only 11 
states have aligned their state high school assessments with postsecondary 
education (Education Week, 2007). Brown and Conley (in press) reviewed the 
alignment of the test content from 60 high school state assessments in 20 states 
and the knowledge and skills needed for university success and found modest 
but uneven alignment. While this study investigated the alignment between 
high school test content and the necessary knowledge and skills for success at 
top research universities across the country, it did not address the preparation 
needed for success in entry-level courses at open access community colleges, 
nor did it include high school exams from California. To gain insight into the 
potential effect of subject matter discordance on why such large numbers of 
California high school graduates need remediation in math and reading once 
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they arrive at the community college campuses, this study of the alignment 
between the content of state mandated high school examinations and the 
content of placement tests used by the community colleges was undertaken. 

Purpose of This Study 

The purpose of this study is to investigate the degree of alignment between 
the content of an array of placement examinations used in community 
colleges and key California Standards Tests (CSTs) in use in California high 
schools. This effort is similar in many ways to much of the content alignment 
work that has taken place regarding specific test content and university- 
level content standards (see for example. Brown & Conley, in press; Conley, 
2003a). It differs, however, in one important way. The placement assessments 
in this study are not being reviewed to determine their alignment with the 
knowledge and skills expected for future university success. Rather, these 
placement assessments are to be analyzed to determine their alignment with 
the specific content that students are expected to have already mastered in 
high school. More specifically, this study seeks to determine the degree of 
content alignment between the "de facto standards" needed for community 
college preparedness (as measured by a plethora of placement exams in use 
across the state) and the standards measured by the augmented CSTs in math 
and English language arts in high school. 

It is important to note that this study is focused on content alignment of 
assessments used at the secondary and community college campuses, not on 
other issues that are also important for understanding the disjunction between 
these educational systems. Other factors such as establishing proficiency 
standards, communication between segments, inconsistency across campuses 
with respect to testing practices, the multiple pathways students undertake as 
they progress through community colleges, English proficiency development, 
and the self-monitoring of assessment systems, while important, are not the 
focus of this investigation. 

Impara (2001) recently detailed several models for conducting content 
alignment studies, ranging from low to high levels of complexity. While 
studies of content alignment have been undertaken for many years, a 
focus on different models for conducting such studies is a more recent 
issue in educational research. A few of the alternative models in content 
alignment research warrant discussion. In a fairly uncomplicated form. 
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an alignment methodology may simply employ content experts to make 
ratings regarding the match between a test item's content and the content 
standards test specifications. Generally, rating scales are employed to evaluate 
the breadth of content coverage and/ or the depth of content coverage 
(cognitive complexity). Frequently, the level of agreement among the raters 
is determined and presented along with the level of content match between 
the assessment and the content standards or test specification. A variation 
of this approach was utilized by Le, Hamilton, & Robyn (2000) in their 
alignment among secondary and postsecondary assessments in California. 
This approach is certainly viable for the current context; however, it is limited 
in the amount of information that it provides. 

A more moderately complex approach is described by LaMarca, et al. 
(2000). This approach enhances the simpler model by broadening the focus 
to include not only content match and depth of coverage, but also balance of 
coverage, performance match, and accessibility. In this approach, accessibility 
refers to a broad enough range of item difficulties on the assessment to allow 
students of all achievement levels to demonstrate their level of knowledge. 

As such, assessments targeted to extremely high (e.g., advanced placement) 
or extremely low (e.g., minimum competency) levels of performance would 
provide less accessibility. 

The most complex of the approaches described by Impara (2001), created 
by Webb (1997, 1999), features multiple categories of criteria for judging 
alignment. Impara states, "because Webb's conceptualization of alignment 
is comprehensive and extends far beyond the two previously described 
models, it can be adapted for use in virtually any context" (p. 4). As a 
result, this approach, with a few modifications, is the procedure selected for 
aligning community college placement exams with the California high school 
assessments. 

Methodology 

This study is comprised of two distinct phases. In the first phase, placement 
exams in widespread use across the California Community College System 
were identified for inclusion in the study. These assessments were then 
subjected to a content analysis methodology to reduce this collection of test 
elements to a "corpus" of community college placement objectives. This 
approach was taken because there are a vast variety of placement tests used 



ILL 



High School and Community College Assessments 



in community colleges across the state and by looking at the most prevalent 
ones used and the material they cover collectively, we get a better picture of 
the diverse expectations students face upon entering the community college 
system. 

The second phase of this project utilized data from the content analysis 
phase in an alignment rating workshop in which subject matter experts made 
judgments about the degree of alignment between the corpus of community 
college placement objectives (CCC objectives) and elements addressed by 
the augmented California Standards Tests (CST elements) given in California 
high schools. The augmented CST tests used in this study included the 
Algebra 11 and Summative High School Math test in mathematics, and the 
Grade 11 test in English language arts, which are part of the Early Assessment 
Program (EAP) which is a program developed by the California Department 
of Education, State Board of Education, and the California State University 
(CSU) system "designed to determine students' readiness to do college- 
level work in English language arts and/ or mathematics while they are in 
their junior year of high school, and to align the CSU placement standards 
with the K-12 standards in English language arts and mathematics" (Early 
Assessment Program Prequently Asked Questions, 2006). These tests were 
chosen for inclusion in this study because they reflect the subject matter 
material students are expected to master in the latter years of high school (i.e., 
California standards for 11th grade) as well as material deemed important 
for placement decisions at campuses within the California State University 
system. Each of the mathematics exams are course specific, whereas the 
nonaugmented form of the English language arts test is given to all 11th grade 
students in California public high schools. Each phase of the project will be 
described in detail in the following paragraphs. 

Phase I - Content Analysis oe CCC Placement Exams 

As mentioned, review of placement test usage at the community college 
campuses in California revealed a vast assortment of placement tests in use 
across the state. However, most of the placement testing was conducted 
with a small subset of placement exams (see Table 6). The most commonly 
used assessments were those that comprise the ACCUPLACER CPTs group 
of tests created by the College Board and the Compass battery of computer 
adaptive tests developed by American College Testing (ACT). In addition, 
in the area of mathematics, the Mathematics Diagnostic Testing Program 
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(MDTP) assessments developed by faculty from the University of California 
and California State University systems were widely used. Similarly, in 
English language arts, the California Test of English Placement (CTEP), 
which was developed by community college faculty, was frequently used 
on the community college campuses. Prom this group of tests, a total of 16 
tests (7 in ELA, 9 in mathematics) were content analyzed. Por the computer 
adaptive tests (various CTP/ACCUPLACER and ACT/COMPASS tests), 
it is not possible to evaluate specific test forms since the items presented 
to each student differ depending on the student's estimated ability level. 

Rather, the confidential test specifications and test blueprints for several tests 
were analyzed. These included three ACCUPLACER/CPT tests in math 
(Arithmetic, Elementary Algebra, College-Level Math), two in language arts 
(ACCUPLACER CPT Reading Comprehension and ACCUPLACER/CPT 
Sentence Skills) along with four ACT / COMPASS math tests (Numerical 
Skills /Pre- Algebra, Algebra, College Algebra, and Geometry) and two ACT/ 

COMPASS tests in language Table 6. 

Approximate Number of Placement Tests Administered by Test Type Analyzed 

Community College Placement Test No. of Administrations 

Mathematics 

CPT Arithmetic, Eiementary Aigebra, College-Levei Math 200,865 

MDTP Algebra Readiness and Elem. Algebra 1 02,779 

ACT Compass Numerical Skills/Pre-Algebra, Algebra, College Algebra, Geometry 57,708 

English Language Arts 

CPT Reading Comprehension, Sentence Skills 327,1 26 

CTEP Sentence Structure and Grammar, Sentence and Syntax Skills 1 03,81 1 

ACT Compass Reading Comprehension, Compass Writing Skills 90,1 62 

CTEP Reading Comprehension 68,898 

NOTE: Data provided by the California Community Colleges Chancellor’s Office Web site regarding 
students enrolled during 2005-2006 academic year (www.cccco.edu). 

Por the MDTP and CTEP 

placement exams, actual test forms were analyzed. Por the adaptive tests, 
content statements from the test specifications and blueprints were merged 
into our content summaries from actual test forms, as described below. 

Phase 1 Content Analysis Participants 

Three analysts evaluated the content of items for placement tests we were 
able to obtain from publishers. Por some tests we could only obtain general 
blueprints, and we merged content statements from those documents into our 
content summaries, as described below. All content analysts had extensive 
previous experience in analyzing the content of curricula, standards, and tests 
and in conducting alignments of tests, curricula, and standards. Two analysts. 



arts (ACT / COMPASS Reading 
Comprehension and ACT / 
COMPASS Writing Skills). 

Access to the secured test details 
was obtained through written 
agreements between the test 
publishers and the National 
Center for Public Policy in 
Higher Education, for whom this 
report is prepared. 
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currently on the staff of the Center for Evaluation, Research, and Student 
Standards at UCLA, have doctorates in educational psychology; the third 
analyst has worked for many years as an alignment specialist for both the 
non-profit and commercial educational markets. 

Training to do the analysis, conducted by the lead content analyst for 
the project, consisted of reviewing examples of items from the tests to be 
analyzed, discussing how the content of each item should be described, then 
individually evaluating the content of several sample items. To determine the 
right level of generality for describing content, we reviewed both California 
state high school standards and community college standards statements put 
together by Achieve. As an example of the level of generality used, the content 
description for a reading comprehension item typically included a statement 
about reading and understanding a certain piece of text, e.g., "Read and 
comprehend a one-paragraph informational text," in addition to a description 
of the particular response skill required by the item, e. g., "Choose the best 
title for a one-paragraph informational text." 

For example, a reading comprehension item read as follows: 

In line 26 (of the passage), impediment means 

A. obstacle 

B. asset 

C. opportunity 

D. defect 

An analyst described the content of this item as follows: 

• Read and understand grade-level material. 

• Determine the meaning of a word in a written passage. 

• Find a synonym for a word. 

In some cases, we also described the skills necessary to identify incorrect 
answers; this happened, for example, when an item required identification of 
an incorrect or untrue statement. Another instance of items requiring students 
to recognize incorrect answers were the cloze (fill in the blank to complete a 
sentence) items used in language arts testing. 
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During the initial phase of training, after individual evaluation of a 
sample item, individual descriptions were compared and discussed, and a 
"correct" consensus evaluation determined. The basic steps of individual 
evaluation and group discussion of items were repeated until the analysts 
were able to agree on the content of several consecutive items. After training, 
the analysts divided up the tests to be analyzed and began their analysis. 
Reliability of the content analyses across analysts was then checked using 
procedures described in the next section. 

Reliability Checks 

To check the consistency of content analyses by different analysts, several 
items from each type of test were evaluated by two analysts and the results 
compared. For a given item, each descriptive content statement (or descriptor) 
for one analyst that closely resembled a descriptor generated by the other 
rater was counted as a match. A descriptor by either rater that did not closely 
resemble any descriptor by the other rater, was counted as a mismatch. We 
coded the following two statements, for example, as being in agreement: 

Analyst 1 : Choose the best title for a one-paragraph informational text. 

Analyst 2: Determine the most suitable title for an informative 
paragraph. 

We coded the following two statements, however, as a mismatch (because 
of the reference in the first statement to identification of information not in the 
paragraph): 

Analyst 3: Read a paragraph and determine specific information that is 
and is not in the paragraph. 

Analyst 2: Identify ideas presented in a passage and use them to answer 
a question related to the passage. 

When conducting the reliability checks, several additional conventions 
were observed. First, if one analyst had combined two skills, such as, "Find 
the meaning of a word in a written passage and find a synonym," while 
another had split these into two separate skills, a separate agreement rating 
was assigned for each part — resulting in two instances of agreement in this 
example. In cases where one analyst described a more general skill, and 
another recorded that general skill in addition providing detailed descriptors 
of the skill that went beyond the level of generality originally agreed upon. 
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we counted the analysts' statements as matching on the general level. Because 
many items required multiple descriptors to capture their content, a single 
item could potentially involve several descriptor matches and/or mismatches. 

To check reliability in math, 20 items were evaluated by two analysts, 
producing a 95% level of agreement between the analysts: Out of 43 possible 
descriptor matches for the 20 items, there were 41 matches and 2 mismatches. 
Given this high level of agreement, the remaining math items were analyzed 
by one person. In reading and language arts, 20 items (10 reading and 
10 language arts) were also analyzed by two persons. Out of 48 possible 
descriptor matches for these items, there were 41 matches and 7 mismatches, 
for an overall agreement level of 85%. The remaining items were analyzed 
by one person. In many cases the mismatches found in reliability checks 
were a result of one rater neglecting to include for one or two items a general 
descriptor that occurred across a broad set of items (e.g., recognize correct 
sentences); this type of error of omission had no effect on the final content 
summary, since descriptors were not repeated in that summary. 

Summarizing the Content 

After the content of the various tests for each subject area (reading, language 
arts, and mathematics) had been analyzed, the content for each area was 
compiled into a single list. To accomplish this, we first generated category 
headings to capture the different types of content across tests in the subject 
area. Under each category heading we then assembled relevant content 
descriptors from each test, in addition to descriptors taken directly from 
the blueprints of tests for which we could not obtain items. The purpose of 
organizing content in this way was to simplify the task of matching content 
statements to those in the high school test documents; an alignment rater 
could look for content under headings such as "percents and decimals" or 
"punctuation," rather than having to scan through a long list of unordered 
content descriptors. After compiling descriptors, we eliminated redundant 
content (that is, content descriptors that used similar or identical language 
to capture the same skills or elements of knowledge) to arrive at the most 
concise list of content for each subject area. 
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Phase II - Alignment Rating Process 



Alignment Workshop Participants 

Community college and university faculty were recruited to serve as 
participants in the alignment rating workshop. All raters had experience 
with state high school assessments or direct involvement teaching entry-level 
community college courses. Training consisted of practice sessions in which 
training items were rated and discussed. From these training sessions, scoring 
criteria and decision rules were developed and refined by the researchers, 
then applied consistently during actual ratings. No fewer than four raters 
rated each test. The ratings for mathematics were conducted by a mixture of 
four community college and university faculty and the ratings for English 
language arts were conducted by a group consisting of five community 
college faculty from the southern California region. 

Alignment Rating Activities 

The alignment rating workshop involved three rating activities. The raters 
first reviewed and scored each of the CCC standards on a five-point scale 
adapted from Marzano (2001) to determine the depth of knowledge of each 
standard. The points of the scale were Retrieval, Comprehension, Analysis, 
Utilization, and Goal Setting/ Monitoring. Marzano's scale is designed to 
reflect increasing cognitive complexity, more sophisticated uses of knowledge, 
and progressively higher levels of meta-cognihve functioning. The raters 
then used the five-point scale to assign a depth-of-knowledge rating to each 
augmented CST assessment element within a discipline. 

The third rating task required raters to review each augmented CST 
element to determine whether it addressed one or more of the CCC objectives. 
Each augmented CST element was rated against each CCC standard and a 
determination was made as to whether that element addressed each CCC 
objective. In that way, an augmented CST element may be linked to any 
number of CCC objectives depending on its content coverage. The result 
of these three analyses was a matrix for each rater (CCC objectives by 
augmented CST assessment elements) that identified which augmented CST 
element addressed which CCC objective, how many CCC objectives within 
a content area were addressed by how many high school test elements, 
and which CCC objectives were not addressed by any augmented CST 
elements. That is, for each CCC content grouping, the number of augmented 
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CST assessment elements addressing it, as well as the number of objectives 
addressed by specific assessment elements could be identified. From these 
ratings, several alignment values are then calculated and compared to 
recommended criterion values from prior alignment research (Webb, 1997, 
1999). 

Alignment Criteria Values 

This process of content focus alignment (Webb, 1997, 1999) yields scales to 
which criterion values are applied to reach overall conclusions about how 
well the corpus of CCC standards align with high school test content. The 
four alignment criteria analyzed were categorical concurrence (the number 
of matches between CCC objectives and augmented CST content), depth 
of knowledge (the cognitive complexity of test elements in relation to CCC 
objectives), range of knowledge (the spread of test content across CCC 
objectives), and balance of representation (the ways in which test elements 
that matched a CCC objective distributed onto CCC objectives within a CCC 
content area grouping). 

Categorical Concurrence 

Raters first determined categorical concurrence, the match between the CCC 
objectives and each element from the high school assessments. The goal is to 
determine two things: First, do the assessments cover areas deemed important 
to community college success as measured by placement exams; and, second, 
are the CCC objectives in their current format useful for determining such 
relationships? Raters were asked to identify which (if any) CCC objectives 
were addressed by each augmented CST element. These ratings were then 
summarized across raters to determine the average number of elements for 
each augmented CST assessment addressing one or more CCC objective. This 
criterion could range from 0 to the total number of elements included on the 
high school test. The recommended benchmark for this alignment criterion 
is that at least six items on average are aligned to each CCC grouping of 
objectives (Webb, 1999). 

Depth of Knowledge Consistency 

Recall that each augmented CST element and each CCC objective was given 
a depth of knowledge rating by each rater. Where categorical concurrence 
was established, the depth of knowledge raters between these components 
was compared. If the augmented CST element was rated at the same or 



2 ^ 



High School and Community College Assessments 



higher level of cognitive complexity as the CCC objective with which it 
corresponded, a relationship was noted. Keeping with Webb's methodology, 
for this analysis the depth of knowledge consistency is measured as the 
percentage of matches between augmented CST elements and a CCC 
objectives wherein the augmented CST element had an equivalent or higher 
depth of knowledge rating than the corresponding CCC objective. Values for 
this criterion range from 0 to 1 .0, with a recommended benchmark greater 
than or equal to .50 (Webb, 1999). That is, for the augmented CST assessment 
to be considered adequately aligned to the community college objectives with 
respect to depth of knowledge, for at least half of the augmented CST to CCC 
matches the augmented CST component should be at or above the cognitive 
complexity level of the corresponding CCC objective. 



Range of Knowledge 



Range of knowledge was determined by tallying the number of CCC 
objectives that were addressed by one or more elements on the high school 
assessment. This criterion gives an estimate of the breadth of knowledge 
addressed by the high school tests. Webb describes the range-of-knowledge 
criterion as a means "to judge whether a comparable span of knowledge 
expected of students by a standard is the same as, or corresponds to, the 
span of knowledge that students need in order to correctly answer the 
assessment items /activities" (Webb, 1999, p. 8). This metric is created by 
calculating the percentage of objectives within a standard addressed by the 
assessment items for each rater. These rater estimates are then averaged 
across raters for a summary range of knowledge value. This criterion's value 
ranges from 0 to 1.0, with a recommended 
benchmark greater than or equal to .50. For 
the augmented CST tests to be considered 
adequately aligned to the CCC objectives 
with respect to range of knowledge, at least 
half of the objectives within a CCC grouping 
should be addressed. In this study, the 
number of objectives for a given grouping 
varied from 3 to 71 in ELA. The ELA content 
groups and the number of objectives in each 
are displayed in Table 7. These include: 

Sentence Structure, Grammar, Syntax, 
and Usage; Punctuation; Rhetorical Skills; 



Table 7. 

Number of CCC Objectives by Content Grouping 

Community College Content Area No. of Objectives 

Language 

Sentence Structure, Grammar, Syntax, and Usage 71 

Punctuation 1 9 

Rhetorical Skills 3 

Organization 5 

Style 7 

Reading 

Vocabulary and Sentence Relationships 4 

Literal Comprehension 1 2 

Main Ideas 4 

Supporting Ideas 4 

Inferences 19 

Applications 3 
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Mathematics 

Whole Numbers and Fractions 

Decimals and Percents 

Applications and Interpreting Tables/Graphs 

Integers and Rational Numbers 

Algebraic Expressions and Operations 

Operations with Exponents 

Equations, Inequalities, and Word Problems 

Functions 

Trigonometry 

Geometry 

Graphing 

Applications and Other Algebra Topics 



Community Coiiege Content Area 



Table 8. 

Number of CCC Objectives by Content Grouping 



No. of Objectives 



20 

16 

14 
6 
16 
8 
11 
9 
8 
16 
16 

15 



Organization; and Style. In Reading, the 
content groupings include: Vocabulary 
and Sentence Relationships; Literal 
Comprehension; Main Ideas; Supporting 
Ideas; Inferences; and Applications. The 
number of objectives in the math content 
groupings ranged from six to 20 in the 
following content groupings: Whole 
Numbers and Fractions; Decimals and 
Percents; Applications and Interpreting 
Tables/Graphs; Integers and Rational 
Numbers; Algebraic Expressions and 
Operations; Operations with Exponents; 
Equations, Inequalities, and Word 



Problems; Functions; Trigonometry; Geometry; Graphing, Applications and 
Other Algebra Topics. 

Balance of Representation 

Balance of representation identified how those items that matched specific 
CCC objectives were distributed onto the CCC objectives they matched. This 
criterion indicates the extent assessment items are evenly distributed across 
objectives. Like the depth of knowledge and range of knowledge criteria, 
balance of representation is expressed via an index score from 0 to 1.0, but 
has a recommended benchmark of .70 rather than .50. This higher threshold 
is established to ensure that reasonably good overlap exists between the test 
elements and the CCC objectives overall. 



Reliability of Ratings 

The depth of knowledge ratings for the CCC objectives and the augmented 
CST content elements obtained in the first two rating exercises were 
analyzed for rater agreement. These analyses utilized generalizability theory 
(Shavelson & Webb, 1991) to investigate the consistency of ratings. The 
generalizability coefficient (g-coefficient) is comparable to the more common 
reliability coefficient from classical test theory and indicates the percent of 
variability in the scores which are due to systematic influences. Relative g- 
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coefficients provide estimates of consistency of rank ordering elements by 
raters, whereas absolute g-coefficients provide estimates of consistency of 
the value of elements across raters. Estimated g-coefficients for the CCC 



ratings were found to be lower than the 




Table 9. 




estimated g-coefficients for the ratings of 


Reliability of Depth of Knowledge Ratings 


augmented CST elements. However, in both 
cases, generalizability coefficients exceeded 
the minimally acceptable threshold of .70 


Subject Area 


Relative 

g-coefficient 


Absolute 

g-coefficient 


Mathematics 

CCC Objectives 


.79 


.79 


and were often closer to or above the more 


Augmented CST Elements 


.91 


.90 


desirable threshold of .80, even for the slightly 
more stringent absolute g-coefficients. 


ELA/Reading 

CCC Objectives 
Augmented CST Eiements 


.73 

.81 


.72 

.79 



Categorical Concurrence 

Ratings of categorical concurrence produced interesting results (see Tables 
10 and 11 for summaries of alignment ratings for EL A and mathematics, 
respectively). In the area of English language arts, we found that the elements 
of the augmented CST Grade 11 EL A test was aligned quite well to all of the 
content groupings for the CCC placement exams, using Webb's criterion of 
having at least six items per grouping to establish sufficient coverage of the 
content. The augmented CST test had the strongest categorical concurrence 
in the area of sentence structure, grammar, syntax, and usage, primarily due 

Table 10. 

Summary of Alignment Ratings for English Language Arts 



Community College Content Area 


Categorical 

Concurrence^ 


Depth of 
Knowledge'’ 


Range of 
Knowledge'’ 


Balance of 
Representation'' 


Language 

Sentence Structure, Grammar, Syntax, and Usage 66.60* 


83.55%* 


86.20%* 


0.92* 


Punctuation 


26.40* 


59.73%* 


100.00%* 


0.99* 


Rhetoricai Skiiis 


15.00* 


49.39% 


100.00%* 


0.92* 


Organization 


16.60* 


90.78%* 


100.00%* 


0.92* 


Styie 


27.40* 


83.14%* 


100.00%* 


0.93* 


Reading 

Vocabuiary and Sentence Reiationships 


13.80* 


70.71%* 


80.00%* 


0.94* 


Literai Comprehension 


68.80* 


93.46%* 


91.67%* 


0.82* 


Main ideas 


14.60* 


100.00%* 


60.00%* 


0.93* 


Supporting Ideas 


14.60* 


87.23%* 


100.00%* 


0.97* 


Inferences 


107.00* 


78.83%* 


91.58%* 


0.84* 


Applications 


14.80* 


92.80%* 


60.00%* 


0.95* 



NOTES: 

^Criterion value for this measure is at least 6.0 
‘^Criterion value for these measures is at least 50% 
‘^Criterion value for this measure is at least 70% 

*lntlicates sufficient values to reflect acceptable alignment 
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to the fact that this CCC content grouping had substantially more elements 
(71) than did the other content groupings. However, even in CCC content 
groupings with a more modest number of elements, there was still substantial 
alignment. In the area of inferences, for example, there are but 19 elements in 
the content grouping, but there was an average of 107 matches between these 
19 elements and the 28 standards comprising the augmented CST test. 

The categorical concurrence results were not as strong in the area of 
mathematics as they were in ELA. Whereas every content grouping in ELA 
showed sufficient alignment on the categorical concurrence metric, the 
same can be said for less than half of the content groupings in mathematics. 
Specifically, the augmented CST tests showed sufficient categorical 
concurrence alignment in the areas of algebraic expressions and operations; 
equations, inequalities, and word problems; functions; geometry; and 
applications and other algebra topics. But they did not adequately align in the 
areas of whole numbers and fractions; decimals and percents; applications 
and interpreting tables /graphs; integers and rational numbers; operations 
with exponents; trigonometry; or graphing. Thus, it appears from a content 
standpoint, there are many more elements addressed by the placement tests 
used at the community colleges than students taking these two high school 
assessments are tested on. It may be that some of these topics like whole 
numbers and fractions or decimals and percents are covered in courses 

Table 11. 

Summary of Alignment Ratings for Mathematics 



Community College Content Area 


Categoricai 

Concurrence^ 


Depth of 
Knowiedge'r 


Range of 
Knowledge‘S 


Balance of 
Representation‘s 


Mathematics 










Whole Numbers and Fractions 


0.00 


N/A 


0.00% 


N/A 


Decimals and Percents 


0.00 


N/A 


0.00% 


N/A 


Applications and Interpreting Tables/Graphs 


2.75 


87.50%* 


8.93% 


0.96* 


Integers and Rational Numbers 


0.75 


100.00%* 


12.50% 


1.00* 


Algebraic Expressions and Operations 


21 .00* 


100.00%* 


71.88%* 


0.85* 


Operations with Exponents 


1.75 


100.00%* 


18.75% 


0.96* 


Equations, Inequalities, and Word Problems 


12.50* 


89.58%* 


59.09%* 


0.84* 


Functions 


11.00* 


79.57%* 


58.33%* 


0.78* 


Trigonometry 


3.75 


100.00%* 


43.75% 


0.96* 


Geometry 


14.75* 


87.75%* 


67.19%* 


0.90* 


Graphing 


4.25 


100.00%* 


25.00% 


0.96* 


Applications and Other Algebra Topics 


6.50* 


92.46%* 


26.67% 


0.83* 



NOTES: 

^Criterion value for this measure is at least 6.0 
‘^Criterion value tor these measures is at least 50% 
‘^Criterion value for this measure is at least 70% 

* Indioates sufficient values to reflect acceptable alignment 
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that precede the courses for which these end-of-course assessments are 
targeted. However, this does not explain the lack of concordance in areas like 
trigonometry or, perhaps, graphing and operations with exponents. 

Depth of Knowledge Consistency 

The depth of knowledge measure of alignment is useful for determining 
whether the elements on the augmented CST exams that are matched to 
objectives from the placement exams have a comparable (or higher) level 
of cognitive demand than do the objects to which they are matched. In the 
case of the English language arts, the augmented CST elements aligned well 
with the placement objectives in all but one content grouping. In rhetorical 
skills, the augmented CST elements were found to have an equal or higher 
level of cognitive demand than the placement objectives for 49.39% of the 
ratings, which is just slightly below the threshold of 50% Webb recommends 
for demonstrating sufficient depth of knowledge alignment. In all other 
content groupings, the values well exceeded this recommended benchmark, 
thereby demonstrating sufficient depth of knowledge alignment in most 
areas of language arts and reading. That is, the high school tests appear to be 
addressing the topics they share with the placement tests at a sufficient level 
of cognitive demand. 

For mathematics, similar results were obtained. With the exception of the 
areas for which no depth of knowledge ratings could be calculated because 
no matches were obtained (i.e., whole numbers and fractions; decimals and 
percents), the high school test content showed sufficient depth of knowledge 
alignment with the community college placement exams. In many cases, all of 
the matches in a content grouping showed that the cognitive demand of the 
augmented CST elements met or exceeded the cognitive demand of the CCC 
placement test objectives. Thus, it appears the depth of knowledge measures 
indicate sufficient alignment between the augmented CST test content and the 
CCC placement tests in both EL A and mathematics. 

Range of Knowledge 

The ratings for range of knowledge differ from the ratings of categorical 
concurrence in that they show not how many matches between the 
augmented CST elements and the CCC objects were obtained, but how many 
of the objectives within a content grouping were matched to an augmented 
CST element. This gives a sense of how broadly covered the CCC placement 
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objects are by the augmented CST elements. In the case of ELA all of the 
content groupings showed sufficient range of knowledge alignment. That is, 
the majority of objectives within each CCC placement test content grouping 
was matched to at least one augmented CST element. Such was not the case 
with mathematics. 

The math tests showed sufficient range of knowledge alignment in 
only four of the 12 CCC placement test content groupings. These included: 
algebraic expressions and operations; equations, inequalities, and word 
problems; functions; and geometry. The augmented CST tests did not show 
sufficient range of knowledge alignment in the areas of whole numbers and 
fractions; decimals and percents; applications and interpreting tables /graphs; 
integers and rational numbers; operations with exponents; trigonometry; 
graphing; or applications and other algebra topics. These results for range 
of knowledge alignment are consistent with the results for categorical 
concurrence in finding that many areas of mathematics that are addressed in 
the placement exams used by community colleges in California are not tested 
by these end-of-course high school assessments, resulting in weak alignment 
across the systems. 

Balance of Representation 

The results for the balance of representation alignment metric show that in 
the area of ELA, the augmented CST elements were adequately distributed 
across the objectives within each CCC placement test content grouping. None 
of the content groupings in either Language or Reading had a balance of 
representation value below the recommended threshold of .70. This suggests 
that the augmented CST elements did not cluster unexpectedly on a few of the 
objectives within a content grouping, but rather they were spread out across 
the objectives within each group. Thus, as with categorical concurrence, depth 
of knowledge consistency, and range of knowledge, it appears the high school 
test content in English language arts is adequately aligned with the content 
of placement exams used in California community colleges with respect to 
balance of representation. 

Good balance of representation alignment between augmented CST 
test content and CCC placement objectives was also found in the area of 
mathematics. As was the case in ELA, all of the content groupings for CCC 
placement tests in mathematics matched to augmented CST content showed 



High School and Community College Assessments 



balance of representation measures in excess of the .70 recommended 
threshold for demonstrating adequate alignment. This suggests that where 
matches between augmented CST content and CCC placement test content 
were observed within a content grouping, the matches were distributed 
across the objectives adequately rather than clustering on a limited number of 
objectives. Thus, the content of the augmented CST high school assessments 
shows good alignment to the community college placement tests with respect 
to balance of representation and depth of knowledge consistency, but falls 
short in some areas with respect to categorical concurrence and range of 
knowledge alignment. 

Conclusions and Recommendations 

The results of this alignment study suggest that the augmented CSTs in 
English language arts demonstrates sufficient alignment with the objectives 
measured by the most prevalent placement exams in use on California 
community college campuses. Results differed across subject areas. Whereas 
the ELA test showed strong alignment in all four areas across content 
groupings, the math tests showed adequate alignment values only with 
respect to depth of knowledge consistency and balance of representation, 
falling short in many content areas in terms of categorical concurrence and 
range of knowledge alignment. Some mathematics topics covered by the 
placement exams are simply not addressed by the augmented CST tests. 

These tended to be either lower-level mathematics concepts such as whole 
numbers or fractions, or they involved topics beyond the level of Algebra 
11, such as trigonometry. This may have resulted from the greater number 
and wider variety of placement exams evaluated in mathematics relative to 
English language arts. 

It is also important to keep in mind that this study analyzed the alignment 
of content from two math tests (augmented versions of the Algebra 11 and 
Summative High School Math assessments) that are part of the Standardized 
Testing and Reporting system for California's secondary education, but are 
not the only tests for 11th grade students in mathematics. These are end-of- 
course exams that are taken only by students taking specific courses, unlike 
the Grade 11 CST in English language arts that is given to all students in the 
11th grade. 

In fact, these math tests are taken by relatively few high school students. 
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In 2006, of 1,731,267 students tested in high school mathematics, only 99,315 
(5.7%) took the Summative High School Math Test and only 213,708 (12.3%) 
took the Algebra II assessment. Most of the students tested in high school 
math are tested on either the Algebra I test (706,416 or 40.8%), the Geometry 
Test (359,802 or 20.8%), or the General Mathematics test (338,766 or 19.6%). 
There is, however, considerable overlap in content between the Summative 
High School Math test and both the Algebra I test and the Geometry Test. 



Not only do relatively few students take the Summative High School 
Mathematics Test and the Algebra II test each year, but those who take the 
test do not perform particularly well on it. According to recent data from the 
California Department of Education (CDE) Web site, less than half (46%) of 
the students taking the Summative High School Mathematics test performed 
at or above the proficient level in 2006, while about 28% performed at the 
Basic level, indicating that about a quarter of the students performed below 
the Basic level of achievement. Only about a quarter of students taking the 
Algebra II test achieve at or above Proficient, with another 27% scoring at the 
Basic level, suggesting that nearly half of all students taking the Algebra II 
test failed to score at least at the Basic level of achievement (see Table 12). So, 
while the test content of these two high school tests may be modestly aligned 

to some topics examined by community 
college placement exams, relatively few 
students are actually taking these tests and 
much fewer are mastering the material they 
cover. As a result, it is unsurprising to see 
a large proportion of students relegated to 
remedial mathematics courses once they 
arrive at the community college campuses. 



Table 12. 

Statewide Proficiency Rates for CST Tests 2006 



CST Test 


% Basic 


% Proficient 


Mathematics 






Algebra II 


27 


25 


Summative High School Math 


28 


46 


English Language Arts 






Grade 1 1 


24 


36 



NOTE: Data obtained from California Department ot Education Web site www.ode.oa.gov. 



The alignment between high school 
assessments and college placement exams is good in English language 
arts, however. So if the tests in ELA are well aligned across high school 
and community college systems, then why do so many students require 
remediation? While alignment between the Grade 11 ELA test in high school 
and the content of community college placement exams appears strong, 
the preparation of high school students in terms of mastering that content 
is seriously lacking. Only 36% of students taking the Grade 11 ELA test in 
2006 reached the level of Proficient or better, with another 24% scoring at the 
Basic level, indicating about two-thirds of the students did not master the 
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material sufficiently to be deemed proficient and roughly 40% of students 
score at a level below Basic achievement. So, while high school students may 
be instructed in and tested on subject matter content that aligns well with the 
expectations at community colleges, alignment alone is insufficient to stem 
the rising need for remediation. These results suggest that alignment between 
high school tests and community college placement examinations may be a 
necessary but insufficient condition to adequately prepare students for the 
transition from secondary to postsecondary education. 

One suggestion for improving the disjunction between high school and 
community college is to make clear to students early in their educational 
careers, perhaps as early as middle school, what is expected of them upon 
enrollment at the community colleges by developing continuity across the 
high school, community college, and four-year college systems. In much the 
same way as the University of California and California State University 
systems have prepared a list of course requirements for admission eligibility, 
community colleges could outline the expected levels of preparation for 
students to undertake college credit-bearing coursework and how that 
relates to the courses they need to take in high school. While creating such 
a set of criteria would require consistency across the community college 
campuses and would be a challenging undertaking, merely establishing 
such expectations is not enough. This information would then need to be 
communicated to students, parents, high school counselors, and high school 
district curriculum personnel on a consistent and ongoing basis. 

One way of sending a uniform message and generating consistent 
expectations across community college campuses is to make common 
the placement testing practices the campuses employ. This might mean 
determining a limited set of existing assessments to be used, or developing 
a placement test battery specifically for the community colleges. Such a 
dramatic shift from existing practices would require involvement at the state 
level, with the Community College Board of Governors playing a major role 
in establishing a consistent and coherent placement testing policy. Additional 
research should be conducted into the alignment of the community college 
placement tests and other existing placements such as the English Placement 
Test and the Entry-Level Math Test in use at the other college systems in the 
state, as well as the alignment between placement tests and the skills actually 
addressed in the community college classrooms. 
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It is also advisable to bring leaders in K-12 education into the discussion. 
The State Board of Education and the Community College Board of 
Governors could work collaboratively to address the problem of aligning 
the systems from both perspectives. Such an effort could have a far-reaching 
impact. Much attention is paid to preparing students for the University 
of California and the California State University systems, either through 
focusing on A-G course requirements, encouraging Advanced Placement/ 
Honors course taking, or emphasizing SAT test performance. It is time we 
make a comparable effort to send clear signals, properly inform, and prepare 
those students who will enroll at the community college campuses, especially 
since they constitute the vast majority of college students in California. 
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Fund for the Improvement of Postsecondary Education: The Early Years (June 2002, #02- 
5). The Fund for the Improvement of Postsecondary Education (FIPSE) attained remarkable 
success in funding innovative and enduring projects during its early years. This report, 
prepared by FIPSE’s early program officers, describes how those results were achieved. 

Losing Ground: A National Status Report on the Affordability of American Higher 
Education (May 2002, #02-3). This national status report documents the declining 
affordability of higher education for American families, and highlights public policies that 
support affordable higher education. It provides state-by-state summaries as well as national 
hndings. 

The Affordability of Higher Education: A Review of Recent Survey Research, by 
John Immerwahr (May 2002, #02-4). This review of recent surveys by Public Agenda 
conhrms that Americans feel that rising college costs threaten to make higher education 
inaccessible for many people. 

Coping with Recession: Public Policy, Economic Downturns, and Higher Education, by 
Patrick M. Callan (February 2002, #02-2). This report outlines the major policy considerations 
that states and institutions of higher education face during economic downturns. 

Competition and Collaboration in California Higher Education, by Kathy Reeves Bracco 
and Patrick M. Callan (January 2002, #02-1). This report argues that the structure of 
California’s state higher education system limits the system’s capacity for collaboration. 

Measuring Up 2000: The State -by -State Report Card for Higher Education (November 
2000, #00-3). This hrst-of-its-kind report card grades each state on its performance in higher 
education. The report card also provides comprehensive prohles of each state and brief states- 
at-a-glance comparisons. 

Beneath the Surface: A Statistical Analysis of the Major Variables Associated with 
State Grades in Measuring Up 2000, by Alisa F. Cunningham and Jane V. Wellman 
(November 2001, #01-4). Using statistical analysis, this report explores the “drivers” that 
predict overall performance in Measuring Up 2000. 

Supplementary Analysis for Measuring Up 2000; An Exploratory Report, by Mario 
Martinez (November 2001, #01-3). This supplement explores the relationships within and 
among the performance categories in Measuring Up 2000. 
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Some Next Steps for States: A Follow-up to Measuring Up 2000, by Dennis Jones and 
Karen Paulson (June 2001, #01-2). This report suggests a range of actions that states can 
take to bridge the gap between state performance identihed in Measuring Up 2000 and 
the formulation of effective policy to improve performance in higher education. 

A Review of Tests Performed on the Data in Measuring Up 2000, by Peter Ewell 
(June 2001, #01-1). This review describes the statistical testing performed on the data in 
Measuring Up 2000 by the National Center for Higher Education Management Systems. 

Recent State Policy Initiatives in Education: A Supplement to Measuring Up 2000, by 
Aims C. McGuinness, Jr. (December 2000, #00-6). This supplement highlights education 
initiatives that states have adopted since 1997-98. 

Assessing Student Learning Outcomes: A Supplement to Measuring Up 2000, by Peter 
Ewell and Paula Ries (December 2000, #00-5). This report is a national survey of state 
efforts to assess student learning outcomes in higher education. 

Technical Guide Documenting Methodology, Indicators and Data Sources for 
Measuring Up 2000 (November 2000, #00-4). 

A State-by-State Report Card on Higher Education: Prospectus (March 2000, #00-1). 
This document summarizes the goals of the National Center’s report-card project. 

Great Expectations: How the Public and Parents — White, African-American, and 
Hispanic — View Higher Education, by John Immerwahr with Tony Foleno (May 2000, #00- 
2). This report by Public Agenda hnds that Americans overwhelmingly see higher education 
as essential for success. Survey results are also available for the following states: 

Great Expectations: How Pennsylvanians View Higher Education (May 2000, #00-2b). 
Great Expectations: How Floridians View Higher Education (August 2000, #00-2c). 
Great Expectations: How Coloradans View Higher Education (August 2000, #00-2d). 
Great Expectations: How Californians View Higher Education (August 2000, #00-2e). 
Great Expectations: How New Yorkers View Higher Education (October 2000, #00-21). 
Great Expectations: How Illinois Residents View Higher Education (October 2000, #00- 
2h). 

State Spending for Higher Education in the Next Decade: The Battle to Sustain Current 
Support, by Harold A. Hovey (July 1999, #99-3). This hscal forecast of state and local 
spending patterns hnds that the vast majority of states will face signihcant hscal dehcits 
over the next eight years, which will in turn lead to increased scrutiny of higher education in 
almost all states, and to curtailed spending for public higher education in many states. 

South Dakota: Developing Policy-Driven Change in Higher Education, by Mario Martinez 
(June 1999, #99-2). This report describes the processes for change in higher education that 
government, business, and higher education leaders are creating and implementing in South 
Dakota. 

Taking Responsibility: Leaders’ Expectations of Higher Education, by John Immerwahr 
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(January 1999, #99-1). This paper reports the views of those most involved with 
decisionmaking about higher education, based on focus groups and a survey conducted by 
Public Agenda. 

The Challenges and Opportunities Facing Higher Education: An Agenda for Policy 
Research, by Dennis Jones, Peter Ewell, and Aims McGuinness, Jr. (December 1998, #98- 
8). This report argues that due to substantial changes in the landscape of postsecondary 
education, new state-level policy frameworks must be developed and implemented. 

Higher Education Governance: Balancing Institutional and Market Influences, by Richard 
C. Richardson, Jr., Kathy Reeves Bracco, Patrick M. Callan, and Joni E. Einney (November 
1998, #98-7). This publication describes the structural relationships that affect institutional 
effectiveness in higher education, and argues that state policy should strive for a balance 
between institutional and market forces. 

Federal Tuition Tax Credits and State Higher Education Policy: A Guide for State 
Policy Makers, by Kristin D. Conklin (December 1998, #98-6). This report examines the 
implications of the federal income tax provisions for students and their families, and makes 
recommendations for state higher education policy. 

The Challenges Facing California Higher Education: A Memorandum to the Next 
Governor of California, by David W. Breneman (September 1998, #98-5). This memorandum 
argues that California should develop a new Master Plan for Higher Education. 

Tidal Wave II Revisited: A Review of Earlier Enrollment Projections for California Higher 
Education, by Gerald C. Hayward, David W. Breneman, and Leobardo E. Estrada (September 
1998, #98-4). This review finds that earlier forecasts of a surge in higher education 
enrollments were accurate. 

Organizing for Learning: The View from the Governor’s Office, by James B. Hunt Jr., chair 
of the National Center for Public Policy and Higher Education, and former governor of North 
Carolina (June 1998, #98-3). This publication is an address to the American Association for 
Higher Education concerning opportunity in higher education. 

The Price of Admission: The Growing Importance of Higher Education, by John 
Immerwahr (Spring 1998, #98-2). This report is a national survey of Americans’ views on 
higher education, conducted and reported by Public Agenda. 

Concept Paper: A National Center to Address Higher Education Policy, by Patrick M. 

Callan (March 1998, #98-1). This concept paper describes the purposes of the National Center 
for Public Policy and Higher Education. 
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